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A  criterion  for  convergence  of  Weak  Greedy  Algorithms1 


V.N.Temlyakov 

University  of  South  Carolina,  Columbia,  SC,  USA 


1.  Introduction 

This  paper  completes  the  investigation  of  necessary  and  sufficient  conditions  on 
the  ’’weakness”  sequence  r  :=  {Ujfcli  for  convergence  of  Weak  Greedy  Algorithm 
for  all  dictionaries  V  and  each  function  (vector)  /  in  Hilbert  space  H.  This  paper 
is  a  follow  up  to  the  papers  [T]  and  [LT],  The  Weak  Greedy  Algorithms  (WGA) 
were  introduced  in  [T].  The  paper  [T]  contains  also  historical  remarks  and  some 
motivation  of  studying  greedy  and  weak  greedy  algorithms.  We  will  not  repeat 
historical  remarks  from  [T]  here  and  refer  the  reader  to  [T]  for  prehistory  of  WGA. 
We  discuss  here  results  on  WGA  in  detail. 

We  remind  first  some  notations  and  definitions  from  the  theory  of  greedy  algo¬ 
rithms.  Let  H  be  a  real  Hilbert  space  with  an  inner  product  (-,  •)  and  the  norm 
||x||  :=  (a:,^)1/2.  We  say  a  set  V  of  functions  (elements)  from  H  is  a  dictionary  if 
each  g  £  V  has  norm  one  (||g||  =  1)  and  span©  =  H.  We  give  now  the  definition  of 
WGA  (see  [T]).  Let  a  weakness  sequence  t  =  {tk}kLi,  0  <  tk  <  1,  be  given. 

Weak  Greedy  Algorithm.  We  define  /q  :=  /.  Then  for  each  m  >  1,  we  induc¬ 
tively  define: 

1 ).  eD  is  any  satisfying 


I  (/m— 1  >  Trn)  I  >  SUp  |  (/^_  j ,  g)  \  \ 

T> 

2). 

/m  :=  fm- 1  -  Um- 1. 

3). 

m 

GTm(f,V ) 

3  =  1 

In  the  case  tk  =  1,  k  =  1,2,...,  we  call  WGA  by  Pure  Greedy  Algorithm 
(PGA).  The  convergence  of  PGA  and  WGA  with  tk  =  t,  0  <  t  <  1,  was  established 
in  [J]  and  [RW].  The  first  sufficient  condition  on  r  which  includes  sequences  with 
limfc^oo  tk  =  0  was  obtained  in  [T]. 

Theorem  A.  Assume 


(1.1) 


OO 


E 
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Then  for  any  dictionary  V  and  any  f  E  H  we  have 

lim  ||/  —  G'fn(f,V)\\  =  0. 

ra— >-oo 

In  [T]  we  reduced  the  proof  of  convergence  of  WGA  with  weakness  sequence  r 
to  some  properties  of  ^-sequences  with  regard  to  r.  Theorem  A  was  derived  from 
the  following  two  statements  proved  in  [T]. 

Proposition  1.1.  Let  r  be  such  that  for  any  e  I2,  a,j  >  0,  j  =  1,  2, . . .  we 

have 

n 

lim  inf  an  )  a?-  /tn  =  0. 

n—foo  '  ^ 

3= 1 

Then  for  any  H ,  V,  and  f  e  H  we  have 


lim  ||/, 


T  I 

m  I 


0. 


Proposition  1.2  (Lemma  2.3,[T]).  If  r  satisfies  the  condition  (1.1)  then  r  sat¬ 
isfies  the  assumption  of  Proposition  1.1. 

The  following  simple  necessary  condition 

OO 

=  °° 

k=  1 


was  mentioned  in  [T],  The  first  nontrivial  necessary  conditions  were  obtained  in 
[LT].  We  proved  in  [LT]  the  following  theorem. 

Theorem  B.  In  the  class  of  monotone  sequences  r  =  {tk}(^=1,  1  >  G  >  ^2  >  ■  ■  ■  > 
0,  the  condition  (1.1)  is  necessary  and  sufficient  for  convergence  of  Weak  Greedy 
Algorithm  for  each  f  and  all  Hilbert  spaces  H  and  dictionaries  V. 

The  proof  of  this  theorem  is  based  on  a  special  procedure  which  we  called  Equal¬ 
izer.  The  generalization  of  that  procedure  plays  an  important  role  in  this  paper 
also  (see  S.3).  In  [LT]  we  gave  an  example  of  a  class  of  sequences  r  for  which  the 
condition  (1.1)  is  not  a  necessary  condition  for  convergence.  We  also  proved  in  [LT] 
a  theorem  which  covers  Theorem  A. 

Theorem  C.  Assume 

00  2S+1-1 

S  tl)1/2  =  oo. 

s=0  k= 2s 

Then  for  any  dictionary  V  and  any  f  G  H  we  have 

lim  ||/  —  Gfn(f,V)\\  =  0. 

m— >-00 

We  prove  in  this  paper  a  criterion  on  r  for  convergence  of  WGA.  Let  us  introduce 
some  notation. 

We  define  by  V  the  class  of  sequences  x  =  {xk}^Li,  Xk  >  0,  k  =  1,  2, . . . ,  with 
the  following  property:  there  exists  a  sequence  0  =  go  <  Qi  <  ■  •  •  such  that 


E 


2s 

A  qs 


<  cx); 


(1.2) 


3 


and 

oo  qs 

(1.3)  E^'E1^00’ 

s=l  k= 1 


where  A qs  :=  gs  -  qs_x. 

Remark  1.1.  It  is  clear  from  this  definition  that  if  x  E  V  and  for  some  N  and  c 
we  have  0  <  <  crfc,  k  >  N,  then  y  E  V. 

Theorem  1.1.  The  condition  r  is  necessary  and  sufficient  for  convergence  of 
Weak  Greedy  Algorithm  with  weakness  sequence  r  /or  eac/i  f  and  all  Hilbert  spaces 
H  and  dictionaries  V. 

Sufficient  part  is  proved  in  Section  2  and  necessary  part  is  proved  in  Section  3. 

2.  Proof  of  convergence 

We  begin  this  section  with  the  following  lemma. 

Lemma  2.1.  Let  {aj}fT1  E  1-2,  a,j  >  0,  j  =  1,  2, -  Then  {an  Y^j= i  ai)^=i  e  T. 

Proof.  Assume  contains  infinitely  many  nonzero  terms  (if  not  the  statement 

is  trivial).  Denote  yn  :=  an  Xo=i  aj  and  define  qs  :=  qs(y )  inductively:  qo  :=  0  and 
for  q0, . . . ,  qs  i  defined  we  choose  qs  as  the  smallest  q  such  that 

(2.1)  (4-9.-1)  E 

n=qs_  i  +  l 


Denote  Qs  :=  {qs-i,qs]-  Then  (2.1)  implies 

2s 


A  qs 


<2-  e  vl  <2_‘E»"' 

n£Qs 


n= 1 


9s 


Thus  it  is  sufficient  to  check  only  (1.3) 

E  2~*E^  <o°- 

s  n= 1 

From  the  definition  of  r/s.  we  have 

(2.2)  E  y„  <  -  l)1,2(  E  !'")1/2<2*- 

n=qs_i  +  l  n=qs_i  +  l 

Next  for  any  N  <  M  we  have 

M  n 

an  = 
n=N  j  =  1  N<j<n<M 
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MM  M 

(2.3)  =  l/2(£  a)  +  <E  W)  >  (£  %)V  2- 

j=N  j=N  j=N 

Combining  (2.2)  and  (2.3)  we  get 


^2  aj  —  ^2  Clj  +  -  2(s+1)/^  +  lla 

i&Qs  j=qs- i+i 


This  implies 

(2.4)  <  C(a)2*A 

3  =  1 


We  have  now 


E2“'Ei£  =  E2“’E  E  ^2E2~’  E  ^ 

s  n=l  s  v=l  n£Qv  s  n£Q3 

2E2~'(E°T  E  “»  <C(o)Ea»<tX3' 

s  j=l  n£Qs  n 

Lemma  2.1  is  proved  now. 

Theorem  2.1.  The  following  two  conditions  are  equivalent 
(C.l)  r  $  V, 

n 

(C.2)  G  I2,  aj  >  0,  liminfa^V^  a7-/tn  =  0. 

J  n— >-00  ^ ' 

J  =  1 

Proof.  We  prove  first  that  (C.l)  =>  (C.2).  Assume  (C.2)  is  not  satisfied:  3{aJ}°E1  G 
I2,  aj  >  0,  such  that 

n 

(2.5)  liminfanV^  aj/tn>  0. 

i=i 

Relation  (2.5)  implies  that  for  some  iV  and  c  >  0  we  have  for  n  >  N  that 

n 

CLn  ^  ^  dj  /  tfi  ^  C 
3  =  1 


or 

n 

tn  —  Cdn  ^  ^  • 

i=i 


5 


This  inequality,  Lemma  2.1,  and  Remark  1.1  imply  that  r  G  V.  The  first  implication 
is  proved  now. 

We  proceed  to  the  second  implication  (C.2)  =>•  (C.l).  Let  r  G  V.  We  construct 
a  sequence  { dj }JT1  G  I2  such  that  for  all  n 


tn  —  CcLn  ^  ' 

3  =  1 


dj 


with  some  C.  This  will  imply  that  (C.2)  is  not  satisfied.  Let  {gs}  :=  {<7s(t)}  be 
a  sequence  from  the  definition  of  V.  We  define  a  sequence  { dj }^T1  as  follows.  For 
nGQs  we  set 

an  :=tn2-sl2  +  2s/2/Aqs. 


Then 

and 

Next, 


ai<2(ti2-  +V(Aq,)-‘) 


Ea»<2j]2-*  E  +  <0°- 

n  s  n£Qs  "  S 

£  a„>  2S/2- 

n£Q„ 


Thus  for  nG  Qs  we  have 


dn  '  dj  E  dn  ^  '  dj  —  tn%  ^ 
3= 1  j&Qs-i 


and 

n 

tn  —  \^dn  ^  '  dj 
3= 1 


for  all  n. 

Theorem  2.1  is  proved  now. 


The  sufficient  part  of  Theorem  1.1  follows  from  Theorem  2.1  and  Proposition 

1.1. 


3.  Construction  of  a  counterexample 

The  following  procedure  which  is  the  generalization  of  Equalizer  from  [LT]  plays 
an  important  role  in  the  construction.  Let  H  be  a  Hilbert  space  with  an  orthonor¬ 
mal  basis  {ej  }|T1.  We  take  two  elements  e,;,  Cj,  i  /  j,  and  define  the  following 
procedure. 

Equalizer  with  schedule  7  :=  {7*,}.  Let  7 &  <  1/5,  /o  :=  e*.  Define: 

(3.1)  gx  :=  axa  -  {1  -  a21)1/2ej;  0:1=71;  (/o,0i)=7i; 

/n  • —  fn—  1  {fn—  1  j  9n)gn'y  Qn  : —  OinCj.  (1  On)  ^  Cj  , 


(3.2) 
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(f ni  9n-\- 1)  —  7n+l;  fn  —  Cln&i  T  bnCj 

We  check 

&n—\  bn  —  1  A  3^ 

to  continue.  If 

un_i  bn— 1  <  3^ 

then  we  take  gn 

:=  2 _1/2(ej  —  Cj)  and 

/n  : —  fn  —  1  (f n—li  9n) 9m 

and  stop  after  this  step.  We  call  this  step  ’’the  final  step”  and  all  other  steps 
’’regular  steps”.  At  each  regular  step  l  we  have 

ai  —  bi  =  ai- 1  —  bi  i  —  7 i(ai  +  (1  —  a2)1/2)  >  ai- 1  —  bi  1  —  21/27 /  >  0. 

After  the  final  step  we  have 

CLn.  bn. 


At  each  regular  step  we  have  by  definition  that 

(fi-i,9i)  =  Ifi- 

At  the  final  step  we  have 

(/n— 1 ) 9n )  =  2~1/2(an_i  -  6n_i)  >  2~1/2(an_2  -  bn  2  -  21/27n_i)  > 

2-1/2(2v/27n_i)  =  27n_i. 

Thus,  if  2yn_i  >  7n  then  the  above  described  Equalizer  is  a  WGA  with  weakness 
sequence  71, . . .  ,7„. 

At  regular  step  l  we  reduce  the  ||  •  ||2  by  y2.  At  the  final  step  we  reduce  the  ||  •  ||2 


by 

We  also  have 
and 

Thus, 

On  the  other  hand 
Therefore, 

and 

(3.3) 


2  (an— 1  bn  1)  <  9yn. 
fln-i  bn— 1  <C  3\/2yn 


n— 1 

un_i  bn—  1  A  1  \/2  ^  '  7j- 

i= 1 

n  — 1 

V2  ^  7j  +  3-\/27n  >  1. 

i=i 

O  —  bi  <  ai- 1  —  bi  1  —  7/. 

n— 1 

0  <  an_i  -  bn  1  <1  -  ^  7/ 


z=i 


n—  1 


Z^7i  <  !■ 


z=i 


In  order  to  apply  the  above  Equalizer  we  need  to  have  the  inequality  2yn_i  >  y„ 
satisfied.  Let  us  use  the  following  regularization  procedure. 


7 


Regularization.  For  a  given  r  =  i,  t  E  Zoo,  we  define  :=  with 

OO 

,R  —  \  9-™  + 

Lk  ■—  /  j  *  Ln+m ■ 

m=0 

Lemma  3.1.  If  r  E  V  fl  then  rR  E  V  H  1^. 

Proof.  Assumption  r  E  V  implies 


(3.4) 


E2  *E<i  <°°- 

s  k= 1 


We  will  prove  that 


9s 


(3.5) 


532-  E(**)2  < 


(X) 


fc=l 


with  the  same  qs  =  qs{T)  as  above.  Thus  (3.5)  will  imply  rR  E  V.  Let  us  prove 
(3.5).  We  have  for  any  N 


N 


N  oo 


N  oo  00 


E<**)2  =  E<E  2-™t„+m)2  = 


fc=i 


00  00 


fc=l  m= 0 

JV 


00  00 


fc=l  m=0  n=0 

JV 


N 


m= 0  n=0 


E  E2_m_”Et‘+"’t‘+“ "  E E2_m_”(EtW)I/2(Eth»)1/2  = 

k=l  m= 0  n=0  k= 1  fc=l 

oo  N  00  00  N 

(  E  =<E  'l+m)1/2)2  <  ( E  =)(  E 2~m  E  *2+m)- 


771  =  0  fc=l 


m.= 0  m.=0  k=  1 


Next, 


and 


JV 


JV 


E  -  E^  + 


m  t 


2 

I  oo 


fc=l 


fc=l 


Therefore  we  got 


and 


oo  N  N 

E  2~m<E  +  ™IMI2=o)  <  2  E  *2  +  CM- 

m= 0  j  =  1  j  =  l 

JV  JV 

E(i?)2<2E*2+cM 

i=i  j'=i 


9s 


9s 


E 2~‘  E(‘f  )2  < 2  E  2~*  E  ^ + cm  <  °°- 

s  fc=l  s  fc=l 

It  is  easy  to  see  that  \\rR  11=0  <  2||r||oo. 

Lemma  3.1  is  proved  now. 
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Thus  for  any  r  G  V  fl  we  have  tr  G  V,  Hr^Hoo  <  2||t||oo,  and 


2Zn_i  >  tn  ,  n  —  2,  3, ... . 


Clearly,  we  also  have  for  all  n 


t  <  tR 

I'n  SI  I'n  ■ 


One  more  restriction  in  the  Equalizer  is  7n  <  1/5.  Define  a  new  sequence  r'  by 

t'n  '■=  min {t*,  1/5}. 

It  is  clear  that  t'gV  and  also  satisfies 

K  1  >  C 

Let  {gs}  :=  {qs(r')}  be  the  sequence  for  r'  from  the  definition  of  V: 


EW<c>0-  E2'E(‘»)2<°°- 

s  ^  s  n= 1 

Let  e  be  a  small  number  which  we  will  specify  later  and  s o  be  such  that 


Eir<f>  £2“*IX)2<‘- 


s>s  o  n=l 


Consider  the  function 


fs0  2  s°^2(ei  +  •  •  •  +  e2so). 


We  have  ||/So||  =  1.  Define 

t'L  :=  max{t/fc,2So+2(AgSo)-1}. 

We  apply  a  mixture  of  Equalizer  with  schedule  {t/}  to  vectors  e*,  «  <  2S°,  and 
the  PGA  to  the  corresponding  residual  of  fSQ.  We  do  this  in  the  following  way.  If 
t"  >  1/5  we  use  PGA  and  throw  away,  say,  2~s°/2e2so  •  If  t'{  <  1/5  we  start  using  the 
Equalizer  with  schedule  {t/}  to  vectors  e\  and  e2so+i-  If  at  some  step  t/  >1/5  then 
we  use  PGA  what  means  throwing  away  one  term  of  the  form  2 ~s°/2ej,  j  G  [1,  2S°], 
Applying  the  Equalizer  to  the  very  last  term  of  the  form  2~s°/2em  we  may  incounter 
with  t'{,  >1/5.  In  such  a  case  we  apply  PGA  and  stop.  As  a  result  we  get 

!,«  +  !■■=  E  4”+1«- 


It  is  clear  that  for  all  k  G  EE+i  we  have 


(40+1)2  <  2~s°~1 


and  also 


^o+i|<2So+1. 
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Assume  that  e  <  1/20.  Then  2So+2(AgSo)~1  <  1/5  and  t'l  >  1/5  is  equivalent  to 
t'k  =  t'k  =  1/5-  If  t'l  <1/5  then  t/  <1/5  and  tk  <  t/.  Therefore,  at  all  Equalizer 
steps  we  have  a  WGA  with  weakness  parameters  {£&}•  If  t'l  =  1/5  we  apply 
PGA  what  is  a  WGA  with  any  tk  at  this  step.  During  this  procedure  which  we  call 
’’working  on  so-level”  we  perform  steps  of  Equalizer  and  steps  of  PGA.  Let 
us  estimate  and  .  It  is  clear  that  M^o  <  2S°.  We  have  applied  the  Equalizer 
to  terms  of  the  form  2 ~s°/2ej  at  most  2S°  times.  For  each  Equalizer  application  we 
have  Yhl j  —  2  (see  (3.3)).  Thus  denoting  E(s0)  :=  {k  :  t'l  <  1/5}  we  get 

Y  t'k  <  2So+1. 

k^E  (so) 

On  the  other  hand  we  have 

y  4  >  m”2,”+2(a9,„)-1 

fcG-E/(so) 


and 

V”  <  Aq,J2. 

Therefore, 

:=  M”  +  <  A9,0/2  4-  2*»  <  A«,„. 

At  each  Equalizer  step  we  reduced  the  ||  •  ||2  by  at  most  9  (t'l)2  2~  80  and  at  each 
PGA  by  at  most  25(t/fc)22~s° .  Thus  the  total  reduction  <5So  for  the  so-level  does  not 
exceed 

25(2-")  +  9(2‘»+4)(A9,„)-1. 

k=  1 

We  are  on  the  (.sq  -f  l)-level  now  and  perform  the  similar  procedure.  We  describe  it 
for  the  general  case  of  an  s-level.  Assume  we  have  after  Ns  \  <  qs-\  steps  of  our 
WGA  the  function  _ 

fs=Y  Ckek 

k£Fa 

with 

(4)2  <  2-,  |F,|  <  2*. 

Define  now 

t'l  :=  maxjt/,  2s+2(Ags)-1},  k  >  Ns_1. 

We  pick  cRkek  with  the  biggest  csk  out  of  { csk ,  k  G  Fs}  and  throw  it  away  if  ^jv  +1  = 
1/5  (we  remind  that  assumption  e  <  1/20  implies  2s+2(A<js)~1  <  1/5)  and  apply 
the  Equalizer  with  schedule  {t'l}  otherwise.  We  continue  to  perform  the  above 
described  procedure  (the  mixture  of  Equalizer  and  PGA  steps)  untill  we  get 

/.+i=  v  « 

k£Fs+ 1 


(c 


l+1)2  <  2 


with 


s  —  1 
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It  is  clear  that  then 
M?0  we  get 

and 

Thus 

and 


.Fs+i|  <  2S+1.  Similarly  to  the  above  estimates  of 

Mf  <  2s 

M™2s+2(Aqs)-1  <  J2  t'k  <  2S+1- 

k£E(s) 

AC  +  M?  <  Aqsj2  +  2s  <  Ags 

Ns  :=  1VS-1  +  +  Mf  <  qs. 


and 


The  total  reduction  Ss  of  the 


2  from  working  on  the  s-level  does  not  exceed 


25(2-)]T(4)2+9(2'+4)(A  q,)-1. 

k=  1 


We  continue  this  process  and  get  that  the  ||  •  ||2  will  be  reduced  by  at  most 

OO  OO  qs  OO 

5^  <  25(  5^  2-s  J2(t'k)2)  +  144  5^  2s (A^)-1  <  169c. 

S=s  0  S=Sq  k=  1  S=S  0 

Choosing  e  small  enough,  say,  e  =  0.005  we  get  divergent  WGA  with  the  weakness 

sequence  r.  This  completes  the  construction  of  the  counterexample. 
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